Gap Analysis: An Innovative Look at Gateway Courses and Student Retention 


5 


Gap Analysis: An Innovative Look at Gateway 
Courses and Student Retention 

William Bloemer, Scott Day and Karen Swan 
University of Illinois Springfield 


Abstract 

In this paper we argue that simply identifying gateway courses in which a large number of 
students fail or withdraw and focusing attention on them may not always be the best use of 
limited resources. No matter what we do, there will always be courses with high D/F/W rates 
simply because of the nature of their content and the preparation of the students who must take 
them. However, some gateway courses defy expectations and produce fewer DFWs than might 
be expected while others produce more. Moreover, the timing of course taking can make a 
difference between success or failure for particular types of students, and failing or 
withdrawing from a course does not always lead to stopping out. In this paper we use examples 
from our work with the analyses of student records to show how one can use student type and 
point in their academic life to predict success in particular gateway courses. Relating 
predictions to observed DFW rates can highlight courses exceeding expectations and those 
which fall below them, and support a more nuanced understanding of where attention is 
needed. Further, we look at whether or not actionable information can be found by looking at 
whether the general connection between DFW’s and attrition holds up for different groups of 
students in specific courses. 
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Introduction 

Improvement in the numbers of young Americans achieving a postsecondary degree has 
been a national priority for over a quarter of a century (Arnold, 1999; Shapiro, Dundar, Yuan, 
Harrell & Wakhungu, 2014), with little improvement seen. Indeed, a myriad of studies in the last 
decades of the 20th century tested the assumptions of theories concerning the reasons why students 
drop out of higher education institutions (Bean & Metzner, 1985; Mallette & Camera, 1991; 
Munro, 1981; Tinto, 1987) to develop models of student progression. 

Likewise, there is a substantial body of literature that has examined determinants of course 
non-completion (Juhong & Maloney, 2006; Ishtani, 2006; Jia, 2014; Montmarquette, 
Mahseredjian, & Houle, 2001; and Wetzel, O’Toole, & Peterson (1999), especially as regards 
online learning (Boston et al., 2009; Clay, Rowland, & Packard, 2008; Morris, Wu, & Finnegan, 
2005; Rovai, 2003). Most recently, learning analytics are being applied to help online educators 
address undergraduate attrition (Baepler, & Murdoch, 2010; Barber, & Sharkey, 2012: Campbell 
& Oblinger, 2007). 
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An emerging strategy for enhancing postsecondary outcomes is to measure the patterns by 
which students reach and move through intermediate stages of degree completion. One of the 
issues identified as contributing to attrition is poor performance in gateway courses (Koch & 
Pistilli, 2015). The Gardner Institute (2015) has identified “gateway courses” as: foundational, 
credit-bearing, lower division courses, for which large numbers of students are at risk of failure 
and which accordingly stand as “gatekeepers” to further study and degree completion. Indeed, 
researchers have found that retention in these courses is strongly correlated with successful degree 
completion (Cabrera, Burkum & La Nasa, 2005; Herzog, 2005; Lewis & Terry, 2016; Moore & 
Shulock, 2009; Offenstein & Shulock, 2010). Koch and Pisitilli (2015) add that “courses with 
high rates of unsuccessful outcomes (DFWI rates) ‘kill’ a student’s grade point average (GPA), 
motivation, and academic progress” (p. 3). 

The problem of gateway courses is especially pernicious in online environments, and 
online educators are attempting to address the issue, primarily through course redesign (Education 
Advisory Board, 2016). At DePaul University, (n.d.), for example, course redesign focuses on 
approaches that will help students learn more effectively. Other strategies for improving success 
in gateway courses include providing extra support for faculty teaching such courses (Nogai & 
Kans) and/or peer support for students taking them (Arendale, 2004). 

These approaches, however, often assume that all gateway courses have the same impact 
on all students. Our investigations indicate that this is not always the case. In this paper we ask 
whether all gateway courses are equally detrimental to student success and/or whether the 
detrimental effects of poor performance in gateway courses apply equally to all students. Answers 
to such questions are important so that institutional resources available for improving courses, 
student support and advising and placement practices can focus on those areas in which the 
problems are most critical. 


Methods 

The challenge is to find the right measures to identify gateway courses, and, once 
identified, to apply the best “fix.” It is simple enough to sort courses in descending order of DFW 
rates, targeting the highest entries for redesign of the course, for changes in its staffing, or for 
providing student support as these are all factors that are under institutional control. This approach 
will often work well enough, but this single, simple measure may overlook some problem cases, 
or, worse yet, identify problems where none exist, causing misplaced or even damaging “fixes.” 

A more careful approach to finding gateway courses would recognize that some courses 
are simply harder than others. There is no single standard for DFW rates that is universally 
applicable. There are also significant differences in students that need to be considered when 
looking at the effectiveness of courses. Students enter with widely different backgrounds and 
learning goals. Individual students change significantly over the course of their studies. The 
effectiveness of a course needs to be judged in the context of the students it serves. Once issues 
have been identified, the response needs to be tailored to those issues. Simply put, it is not 
reasonable to expect all courses to serve all students equally well. Efforts to do so are doomed 
from the start and may actually do harm. More careful matching of students and courses is 
necessary before other “fixes” are applied. 
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Finally, if retention and degree completion are the goal, the effectiveness of the course 
needs to include not only the DFW rates it produces, but the subsequent persistence of the students 
in the course whether they passed it or not. Enrollment and performance in courses needs to be 
tied to persistence measures as well. 

Subjects and Setting 

The data from which out observations are drawn included all undergraduate degree-seeking 
students enrolled in a small, Midwestern public university over a four-year period. The academic 
calendar is semester based. Course grades were those awarded at the end of each term, not the final 
transcript grade which is occasionally different for any number of reasons. This is the grade that 
appears on the students’ report card and is most likely to influence the student’s immediate 
decision to continue their degree studies. 

DFW rates were used to measure students’ success in courses studied. An end of term 
grade of D or F or prior Withdrawal indicates that the student failed to complete the course 
successfully. 

Enrollment in the next regularly scheduled tenn or graduation was used as a measure of 
persistence. This measure was chosen for two reasons. It is known that students who have a break 
in continuous enrollment at this institution are much less likely to graduate. Although there are 
exceptions, of course, most students who leave simply don’t come back. Secondly, connecting 
individual courses to student persistence needs to be done with a short-tenn measure. It is hard to 
attribute a student’s departure to a particular course if they have taken many other courses in the 
meantime. 

For this study, the degree-seeking undergraduate student population was classified into 
four Student Types. First-time freshman versus transfer students was the first distinction. Within 
the first time freshmen group, students who did or did not enter in the Honors program were 
distinguished. The transfer students were further classified according to whether they were majors 
in online programs or on campus programs. The four Student Types, then, were Native Freshmen, 
Honors Freshmen, On-ground Transfers, and Online Transfers. 

Students were also further classified according to stages in their academic life cycle. The 
first term is particularly critical for transfer students and was defined as the first stage. For 
freshmen, the end of the first year is a critical time, so the second term was also defined as a stage. 
The second year, and third year were the next two stages considered. Anything beyond the third 
year was considered to be the last stage. Thus the dozen regular semesters that might make up a 
students’ trek towards a six-year degree completion were broken down into First Tenn, Second 
Tenn, Second Year, Third Year and Later. 

Analyses 

To provide a better context for the interpretation of course DFW rates, a binary logistic 
regression was used to predict the probability that a student would post a D, F, or W grade in any 
specific course using all courses that the institution offered over a particular four-year period of 
time. The predictor variables used were the Student Type and the point in the Student Life Cycle 
(as defined above), plus the most significant individual predictors for each student: their prior 
cumulative GPA, and the fraction of previous courses in which they received D, F, or W grades. 
The latter factor is known to add predictive power over prior GPA alone. There do appear to be 
students with high GPAs who also have high W rates, perhaps to protect those GPAs. The 
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regression basically predicts, then, the chance that an individual student will get a DFW grade in 
an average course. 

The regressed model makes it possible to predict the overall DFW rate for any course, or 
section of a course, based on the characteristics of the students who are enrolled in it. These 
predicted DFW rates can then be used as a benchmark against which actual course performance 
can be measured. Thus, these predicted DFW rates as well as the observed DFW rates were 
recorded for each of the courses studied, and the difference (Gap) between them calculated. 
Courses were then ranked according to observed DFW rates and according to the Gap between 
observed and predicted rates. 

Context is again critically important for connecting individual courses to student 
persistence. The fraction of students who persist is quite different for different types of students, 
at different points in their degree paths. When looking at the impact of any particular event (taking 
a course or getting a particular grade in the course), another kind of Gap was calculated — the 
difference between the persistence rate for a particular group of students in the course being 
examined compared to the persistence rate of students in all courses of the same type at the same 
point in their degree path. 

While all grades posted for the students in this study were used to develop the predictive 
model for course DFW rates and for persistence rates, the courses that were selected for detailed 
analysis were chosen from a particular subset of the undergraduate general education curriculum. 
This particular subset of general education courses is related to the institution’s specific mission 
and values, and is required of all undergraduate students. The structure of the requirement gives 
students great flexibility in term of the specific courses taken and their timing. This adds degrees 
of freedom in this part of the curriculum that are often not present in highly structured majors, 
adding placement and advising to the list as realistic options to improve retention related to these 
specific courses. 


Results 

Table 1 shows notable cases that result from the analysis of the gap between predicted and 
actual DFW rates for the 34 general education courses considered here. DFW rates are given in 
percentage of students getting a D, F, or W in each course. 

As expected, the ranking of courses based on actual and predicted DFW rates often told 
similar stories, as can be seen in the first three cases presented. There were also, however, some 
significant differences. Some courses with DFW rates high enough to trigger alarms, also had 
high predicted DFW rates. When the gaps between observed and predicted DFW rates were 
considered, the courses receded out of the alarm range. The sixth course in Table 1 moves from 
being in the top five problematic courses by actual DFW rate to the middle of the pack when 
judged by the gap. Indeed, Table 1 shows that some courses are just living up to predictions, but 
that in some cases, such as this one, the courses in question are performing better than expected, 
given the population they serve. This can be seen most clearly in the seventh course in Table 1. 
While in the top ten of problem courses by actual DFW rates, the gap between actual and predicted 
DFW rate is large and negative. It is actually performing better than expected for the student 
population being served. Efforts to “fix” such a course are at best misdirected and potentially even 
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harmful. Such a course may be worthy of further scrutiny, not to improve it, but rather to replicate 
its relative success with a specific problematic student population. 


KanK ny 
%DFW 

%DFW 

Predicted 

Gap 

Rank by Gap 

1 

30.20% 

22.40% 

7.90% 

1 

3 

26.70% 

19.40% 

7.20% 

2 

2 

28.00% 

21.60% 

6.40% 

3 

6 

18.30% 

13.90% 

4.40% 

4 

13 

13.90% 

10.70% 

3.20% 

5 

5 

19.60% 

20.90% 

-1.30% 

15 

9 

16.40% 

22.90% 

-6.50% 

26 

33 

1.70% 

12.70% 

-11.00% 

30 

34 

0.00% 

14.80% 

-14.80% 

34 


Table 1. Effect of Gap Analyses to Identity Gateway Courses for Notable Cases: 
Courses Ranked by Highest to Lowest DFW Rates and by Size of Gap 
(negative numbers indicate courses performing above expectation). 


There are also courses with DFW rates that are not high enough to attract attention, but are 
much higher than would be expected for their given student population. Exploring the gap between 
observed and predicted DFW rates brings such courses that might otherwise be overlooked into 
consideration. The fifth row in Table 1 shows a course that might be overlooked using actual 
DFW rates. Using that gap as the measure brings it into the top 5 of courses that need to be looked 
at. 

Finally, there are courses with extremely low DFW rates, perhaps even rates of zero, 
despite the fact that the predicted DFW rates for enrolled students are substantial as seen in the 
last two rows of Table 1. These might be particularly effective courses whose structure and 
methods need to be replicated. They might also be courses whose perfonnance is more a cause 
for concern than celebration. 

The connection between specific courses and student persistence was also examined and 
broken down by student type and point in academic life cycle. Again, this classification of student 
persistence does not produce useful infonnation for all courses, but there are occasionally findings 
that are genuinely surprising and useful. Table 2 shows the results of a typical analysis of this sort. 
The data is presented for students who simply took this course compared with similar students who 
didn’t take it. The same approach can be used, however, for students receiving a D, F, or W versus 
students passing the course or other criteria of interest. The columns in the table are being used to 
classify students by type. The rows break out students at various stages of their degree paths. Each 
cell shows the difference between the persistence of the group of students taking this course 
compared with those who didn’t take it. A positive value indicates the percentage by which the 
students of interest were more likely to persist. A negative value indicates the percentage by which 
students were less likely to persist. 
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Persistence 

Gap 

Native 

Freshman 

Honors 

Freshman 

On-ground 

Transfer 

Online 

Transfer 

first term 

* 

* 

-18.9% 

-15.0% 

second term 

* 

* 

0.3% 

5.9% 

second year 

9.0% 

4.6% 

-4.0% 

6.1% 

third year 

3.8% 

2.7% 

0.7% 

-3.9% 

later 

-1.7% 

0.8% 

2.4% 

15.4% 

numbers are 

too small to make sense in this 

course 



Table 2. Connecting a Course to Near Term Retention by Student Type and Level for Students 
Enrolled in a Particular Course. 


In the case of the particular course shown in Table 2, it was found that taking the course 
(regardless of the grade earned) was generally associated with average persistence. When the 
classifications are removed, students who took this course persisted 1.2% more often than similar 
students who did not take it. However, Table 2 does show a real issue for transfer students taking 
the course in their first term, whether or not they were online. First semester transfer students were 
less likely to persist, by 15-20%. If transfer students took the same course in a later term, there 
were virtually no problems with persistence. There might be many explanations for this effect, 
but regardless of the reason, keeping new transfer students out of this particular course would be 
an obvious, and easily accomplished step toward higher retention. 

Another course with average DFW rates produced surprising results when connected to a 
measure of short term student persistence. Students with a DFW grade in this course did not show 
the expected drop in persistence of about 18%, but rather persistence fell by only 2%. What might 
be going on in that course to account for this is not going to be found in the database, but further 
investigation is surely warranted as it appears that something very good was occurring in it that 
might be replicated in other courses. That there may be courses which serve as “guardrail” courses 
associated with increased student persistence is an intriguing possibility. 

Along those lines, we found that the very act of enrolling in this particular kind of general 
education course was associated with increased persistence across student types and levels. While 
it is tempting to think that the curriculum is resonating well with students, the probable explanation 
is likely more mundane. The courses investigated fulfill an institution specific general education 
requirement that reflects particular institutional values. Students who intend to transfer to other 
institutions to complete their studies need not be concerned about them. If anything, enrollment 
is this group of courses is a confirmation of the students’ intentions to complete their degree here. 
Finding signals that reveal students’ true degree plans can also be helpful in targeting retention 
efforts. 
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Discussion 

Gateway courses with high DFW rates contribute directly to student attrition and are an 
obvious place to direct retention efforts. The identification of problem courses benefits from the 
inclusion, however, of some measure of the expected DFW rates for each course. The approach 
in the research reported here was to base expectations on student type and point in their degree 
path, as well as significant predictors of individual student perfonnance. Other approaches might 
be more appropriate for other institutions based on local factors impacting student success. 

In any case, deviations from projected DFW rates can be used to identify specific courses 
particularly in need of help as well as courses in which students are doing better than expected that 
could perhaps be mined for strategies that increase student success. Finding courses that are 
working well can’t help but inform responses to courses that could be more effective. 

Moreover, connecting specific courses to student persistence for different types of students 
at different points in their degree paths can be used to improve student advising and 
placement. When possible, the quickest, most cost-effective solution to high DFW and attrition 
rates may be to make sure that particular kinds of students are not attempting courses that are 
known to be particularly difficult for students at their current stage of academic career. 

Although the results reported in this paper are clearly limited to the undergraduate 
population at the university studied, they also plainly suggest that the effects of gateway courses 
are mediated by student types and academic stages. Thoughtful examination requires, then, 
adjusting observed course D/F/W rates for the characteristics of the students enrolled. In doing so, 
it is essential to identify courses whose performance issues are real, not apparent, so that 
institutional resources available for improving courses, high quality student advising and 
placement practices can be focused on those areas in which problems actually exist. 
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